A Multi-Stage Algorithm for Text Documents Filtering Based on Physical Knowledge

Submitted: Aug 7, 2013; Accepted: Sep 18, 2013; Published: Sep 25, 2013 Abstract: To make a large amount of text documents available to perception, it is necessary to spread the information sources in thematic groups. In this paper, filtering of electronic documents is based on a preliminary multi-stage clustering algorithm: at the first stage for reducing the feature space Kohonen maps (SOM) are used, at the second stage FOREL algorithm is used to automatically determine the number of clusters. To submit documents in the terms space the "term-document" model has been chosen, since it allows using a possible account of morphology, as it may be applied for "noise" clearing. At the stage of thematic filtration a semantic description of the document is preparing exactly, the document frequency portrait. Subject identification, according developed algorithm, is implemented using the document frequency portrait and the neural network weights. Implemented verification of the developed filtering system effectiveness, showed high accuracy and completeness of the electronic documents thematic filtering.